Skip to content

refactor!: align DAG APIs and share candidate graphs - #480

Open
zzylol wants to merge 33 commits into
refactor/candidate-post-asap-dagsfrom
docs/planner-output-layers
Open

zzylol wants to merge 33 commits into
refactor/candidate-post-asap-dagsfrom
docs/planner-output-layers

Conversation

@zzylol

@zzylol zzylol commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #508. Implements the API names from the layering proposal in #509; this PR's copy of docs/design_docs/proposals/planner-layering.md is byte-identical to #509's head (41076ca).

Why

The architecture's DAG names did not match the Rust APIs. Callers also had to compose a separate lifecycle enumerator, a lifecycle plan kept beside the DAG, and physical candidate wrappers between layers. This PR makes the public pipeline use the named candidate collections from #509 end to end, sharing logical graphs and physical compilation.

What

  • Individual graphs are named PreASAPDAG, LogicalPostASAPDAG, LifecyclePostASAPDAG and PhysicalPostASAPDAG, and a runtime-bound graph is the execution handle PhysicalExecution. Compatibility aliases for the old names are removed.
  • SummaryMaintenanceLifecyclePlan is merged into LifecyclePostASAPDAG: a DAG root with each state's lifecycle, retention and window framework. Per-node timing is derived from it as a LogicalPostASAPDAGAssignment overlay on the shared index. Its error type is now LifecyclePostASAPDAGError.
  • The candidate collections are CandidatePreASAPDAGs (from lower_pre_asap_dag_candidates), CandidateLogicalPostASAPDAGs (logical), CandidateLifecyclePostASAPDAGs (every lifecycle assignment, lazy and budget-checked; from with_timing_for_root or from_post_asap_dag) and CandidatePhysicalPostASAPDAGs (from compile_physical_dag_candidates).
  • Workload and candidate IDs, lifecycle metadata, rejections, timing failures and compilation errors are all preserved. Unknown costs stay unknown. Compatible assignments share one compiled graph, and precompute/query cuts are built on demand.
  • No generation stage selects a winner. Selection is a Planner function over the deployment's cost model; the new layer-4 selection entry point comes in a separate PR. Docs are updated to say so, and the layer-2 name is now "summary lifecycle planning".
  • Docs added or updated: the DAG API alignment plan, the naming table and the breaking-change migration guide.

Before this PR

A deployment used Rc<QueryExpr>, Rc<SummaryNode>, PostAsapDag and CompiledPhysicalDag. It enumerated lifecycle choices itself, held a SummaryMaintenanceLifecyclePlan beside the DAG, attached timing, and assembled physical candidate wrappers.

After this PR

CandidatePreASAPDAGs -> CandidateLogicalPostASAPDAGs -> CandidateLifecyclePostASAPDAGs -> CandidatePhysicalPostASAPDAGs

A caller runs logical.with_timing_for_root(...) and passes the result to compile_physical_dag_candidates(...). It inspects graphs and diagnostics through iter() and gets typed execution cuts with materialize(index). Downstream consumers must migrate to the breaking names (see docs/develop_docs/dag-api-migration.md) before updating their Planner dependency.

Validation

  • cargo test --workspace --no-fail-fast: 1502 passed, 0 failed.
  • cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings pass.
  • Regression tests cover collection handoff, workload entry identity, lazy assignment budgets, unknown costs, graph sharing, compilation reuse that depends on contracts and roots, transport equivalence and error preservation.

🤖 Generated with Claude Code

@zzylol zzylol changed the title docs: define how PlanSpace, PlanOutput, and PostAsapDag relate docs: define Planner DAG names and planning layers Sep 30, 2026
@zzylol zzylol changed the title docs: define Planner DAG names and planning layers docs: define DAG names and preserve candidates through Planner layers Sep 30, 2026
@zzylol zzylol changed the title docs: define DAG names and preserve candidates through Planner layers refactor!: align DAG APIs and share candidate graphs Sep 30, 2026
@zzylol
zzylol changed the base branch from main to refactor/candidate-post-asap-dags September 30, 2026 16:56
@zzylol
zzylol force-pushed the refactor/candidate-post-asap-dags branch from ea12b31 to 763bef3 Compare September 30, 2026 18:27
@zzylol
zzylol force-pushed the docs/planner-output-layers branch from 5307489 to 536dabf Compare September 30, 2026 18:27
zzylol and others added 21 commits September 30, 2026 19:17
#445, #470, #472, and #478 each described the Planner output from a
different angle. Add an output-layers section that places them in order:
candidate space, selected logical plan, and exported logical DAG. Say that
PlanOutput is derived from PlanSpace rather than being a second output.
Use "candidate" instead of "alternative" throughout input-output-workflow.md.

Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al, deployment

Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record the owner-approved layering: logical PlanSpace, summary lifecycle as
the only source of timing, physical compilation cut by timing, and a
deployment that prices lifecycle assignments, supplies data and state, and
executes Planner-compiled DAGs. Link it from the output layers section.

Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use PreASAPDAG, PostASAPDAG, and PhysicalDAG as design names while mapping current Rust APIs. Document frontend lowering, logical candidates, lifecycle-only timing, and physical compilation/cuts.

Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use candidate PostASAPDAGs as the design name and CandidatePostASAPDAGs for the renamed Rust collection. Keep explicit selection helpers separate from the candidate-preserving pipeline.

Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebase note: conflicts in docs/design_docs/architecture/input-output-workflow.md
with the stack below are resolved to this commit's version of the file, as
integration merge e59640f resolved them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Name shared DAG roots and frontend candidate collections; distinguish bound execution graphs from compiled PhysicalDAGs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compile shared logical roots through the existing validator and operator lowering. Enumerate lifecycle assignments without implicit winner selection, retain unknown costs and rejections, and share compatible compiled physical graphs across on-demand timing cuts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Encapsulate lifecycle enumeration in CandidatePostASAPDAGs with timing. Compile that collection directly into CandidatePhysicalDAGs, retaining metadata and failures while sharing compatible compilations. Remove public lifecycle enumerator and physical candidate descriptor APIs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Binding runtime sources is an execution step of a PhysicalDAG, not another
DAG. Rename BoundPhysicalDAG to PhysicalExecution: PhysicalDAG::instantiate
returns it, and it lives for one execution only. The cost model's evidence
graph keeps its own name instead of borrowing the runtime one through an
import alias.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nning

asap-physical-operators depended on asap-aware-mapping only to name the
timed collection's metadata and error types. compile_physical_dag_candidates
now takes any (metadata, Result<PostASAPDAGAssignment, E>) iterator, such as
CandidatePostASAPDAGsWithTiming::iter(), and CandidatePhysicalDAGs<M, E>
keeps the caller's typed timing error in PhysicalCandidateError::Timing
beside PhysicalCandidateError::Compile, instead of a stringified error.
The mapping crate is again only a dev-dependency.

compile and frontier_from_timing take a PostASAPDAGView, replacing the public
PhysicalCompileInput trait. PostASAPDAGIndex projects its node records once;
an assignment's view overlays its timing on them, so compiling, cutting and
validating an assignment no longer re-projects and clones the graph. The
index also replaces PostAsapDagCompilation and
export_post_asap_dag_with_node_ids, and the duplicate node vector is gone.

CandidatePostASAPDAGs is a plain struct again (its roots are read through
roots()); the timed stage is the separate CandidatePostASAPDAGsWithTiming,
so neither needs a stage parameter, PhantomData or Deref. The timed
collection restores lifecycle_guarantee() so a deployment can price an
alternative before binding it. A state with no lifecycle alternative now
yields a NoAlternatives diagnostic entry instead of a candidate that
silently disappears, and every budget overflow reports ExpansionLimit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 10 commits September 30, 2026 19:17
The collection's cuts were only compared with its own shared compilation,
so a collection that ignored its assignments' timing still passed. Compare
each materialized candidate with compile_candidate over the assignment's
transport after a PostASAPDAGDocument JSON round trip, and assert that an
ingestion-time Binary lowers differently from a query-time one. Compiling
the untimed index instead of the assignment fails this test.

Also cover the view's timing overlay, and a state with no lifecycle
alternative: before, its logical candidate produced zero assignments and
vanished (Ok(0)); it now reports NoAlternatives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostAsapNodeId, PostAsapOperatorPayload, PostAsapDagNode, PostAsapDagEdge,
PostAsapDagValidationError, PostAsapNodeIdentityMap, PostAsapSubstitution
and the InvalidPostAsapDag variants now use the PostASAP/DAG casing of the
other graph names. No compatibility aliases; the migration table lists them.
Serialized forms do not contain type names and are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe the two plain candidate collections, the view-based compile, the
caller-typed physical candidate errors, lifecycle_guarantee, and
PhysicalExecution as an execution handle. Remove APIs the docs said were
removed but never existed (compile_timed_candidates, PhysicalDAGCandidate),
the agent instructions in the alignment proposal's baseline, and the
ingestion-time Binary exception from design docs, where it is an
implementation detail (the developer migration guide keeps it).

Restore #485's statement that candidates do not choose placement and #508's
CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the
split test table in physical-planning-and-deployment.md, and take
planner-backend-layering.md verbatim from #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lifecycle_guarantee accepted a lifecycle no state offered; it now names the
state and rejects an unknown state (UnknownSummary) or an unoffered lifecycle
(NotAnAlternative). The logical expansion budget now reports
CandidateTimingError::ExpansionLimit like the assignment budget, instead of a
PhysicalRealization string.

Tests: pricing through the timed collection, including rejected requests;
the logical budget variant; iter() yielding exactly one NoAlternatives entry
for a state without alternatives; typed Compile/Timing matches in place of
is_err(); and every non-Binary payload fixture lowering identically at
ingestion and query time, which shared compilation relies on.
PhysicalExecution's doc now says bind/bind_with_data_sources return it and
that each execute call is a separate run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selection is an ASAPPlanner library function: the deployment supplies
prices, accuracy requirements and capabilities, and Planner selection returns
the optimal plan. Reword passages that had the deployment select, choose or
price-and-pick candidates, assignments or placement. Index the DAG API
alignment and migration docs, update lifecycle_guarantee's signature, and drop
"this branch" wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Precompute versus query timing comes from Planner's lifecycle selection;
the backend only decides where the selected plan runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copies docs/design_docs/proposals/planner-backend-layering.md from #509's
head 3607270 so this file merges cleanly after #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The layering proposal (#509) names a single annotated DAG,
LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The
code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the
timed collection CandidatePostASAPDAGsWithTiming.

- Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its
  error to LifecyclePostASAPDAGError. The type already held the root and
  each state's lifecycle, retention and window framework; per-node timing
  stays derived by execution_assignment as the existing
  PostASAPDAGAssignment overlay, so no timed graph is stored beside it and
  no new type is added.
- Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs;
  CandidatePostASAPDAGs stays the logical collection.
- Call layer 2 "summary lifecycle planning" and reword comments that had
  the deployment choose or rank; selection is Planner's, over the
  deployment's cost model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several docs still said the deployment or "downstream" selects, ranks or
chooses the plan, and the glossary said physical plans are owned by
downstream systems. Per the layering proposal (#509), Planner compiles
and selects the physical plan using the deployment's cost model; the
deployment supplies prices and executes the selected plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#509 carries the proposal; #480's copy now matches its head a095ce5
exactly, so merging either PR first leaves no conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the refactor/candidate-post-asap-dags branch from 763bef3 to 11f4eeb Compare September 30, 2026 20:25
@zzylol
zzylol force-pushed the docs/planner-output-layers branch from 536dabf to c3f10eb Compare September 30, 2026 20:25
zzylol and others added 2 commits September 30, 2026 21:54
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostASAPDAG -> LogicalPostASAPDAG, CandidatePostASAPDAGs ->
CandidateLogicalPostASAPDAGs, PhysicalDAG -> PhysicalPostASAPDAG,
CandidatePhysicalDAGs -> CandidatePhysicalPostASAPDAGs, plus the
PostASAPDAG* compounds and InvalidPostASAPDAG. Serialized forms are
unchanged. Re-copies planner-layering.md from #509 (41076ca).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants